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Abstract-In recent years, with the vigorous development of 
information and communication technology, industrial, 
governmental, academic circles lay more emphasis upon and 
invest more in the E-learning research and its application. The 
E-learning has already taken shape in its technological 
development and organizational operation, which is bringing 
greater added value to itself. However, failures should not be 
neglected. Three correlative problems come up urgently, namely, 
evaluation of the E-learning effectiveness, plan management, test, 
and efficient resources distribution. Through the present 
research, I propose that the governmental E-learning website 
effectiveness evaluation should be basically arranged on four 
levels, which are public policies, training plan, curricula, and 
technological design. Evaluation tools like policy indicators, 
four-level pattern, and ROI (return of investment) also ought to 
be associated as references in order to construct an integrated 
E-learning evaluation frame. The object of the present research 
is Cultural Affairs, School of E-learning, Executive Yuan. The 
findings are not only appropriate to the evaluation of 
governmental website E-learning effectiveness but also 
conducive to evaluation of the same type for enterprises and 
academic establishments or even other organisms. 
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I. INTRODUCTION 

The information and communication technology has been 
vigorously developing since the end of the 20th century. In 
developed countries, all kinds of organizations, from 
governments and academic establishments to enterprises, have 
been investing a large number of resources in the research on 
learning, teaching and training programs with the aim to 
prepare and improve their competitiveness for a century of 
advanced knowledge or the 21st century. Most experts and 
scholars optimistically believed that this vogue of information 
technology and education development would deeply change 
human learning activities, and that, concurrent with the 
lifelong learning and the knowledge economy, it was sure to 
bring about a brand-new learning and training pattern. 

Unfortunately, against their great conception, it was not 
rare that the efforts made by industries, governments and 
academic establishments ended in failure. One of such failures 
was the Fathom.Com, a commercial teaching program 
launched by Columbia University of the U.S.A. And more and 
more failures threaten to eclipse the numerous advantages 
taken on by E-learning and its technological and 
organizational feasibility. Thus, people are commencing on 
the reflection upon the E-learning affectivity and its 
evaluation. 

Recently our Government is promoting National Science 
and Technology Program for E-learning. The ministries and 



councils also construct their E-learning websites and develop 
corresponding systems, platforms, and coursewares for 
educational training as well as for public service. These 
E-learning sites are not purely commercial or profitable and 
the cost is extremely high. Under these circumstances, the 
problem of their effectiveness evaluation and durable running 
is sure to become the focus of attention among legislators, 
administrators, enterprises, and citizens. That is also the 
subject of the present research. 

Ordinary enterprises adopt the Donald Kirkpatrick pattern 
to evaluate the E-learning effectiveness [4]. However, 
governmental policies are different from commercial 
enterprises and economic indicators are not their only 
measuring tools. Through the present research, we propose 
that, according to the roles played by different participants, 
the evaluation of governmental E-learning website 
effectiveness should be basically arranged on four levels, 
which are public policies, training plan, curricula, and 
technological design. On the first level, the E-learning 
belongs to the category of public policies. The American 
scholar of public policies Duncan MacRae does think that the 
indicators for the evaluation of government efficiency should 
be represented by these three "end values": net economic 
benefit, subjective well-being, and equity [1]. So when we 
talk about the effectiveness of governmental E-learning 
websites, we should first and foremost be aware whether 
those three end values have been attained. 

But those end values are not all and they are becoming 
less and less convincing because the situation is no more what 
it was. In recent years, the vogue is the notion of 
"entrepreneurial government". According to this notion, 
public functionaries should serve as public entrepreneurs and 
do everything possible to amplify the government's resources 
[1]. Thus, in respect of the plan management, the efficiency & 
effectiveness of the use that they make of those resources are 
also important connotations. In this sense, the government 
may, as enterprises do, adopt the Donald Kirkpatrick pattern 
and establish its own measuring tools. In respect of 
curriculum and courseware designs, learners' experiences of 
and reactions on E-learning are the focus of attention to 
teachers and technicians, since E-learning involves naturally 
many creative teaching activities and software development. 

In the present research we take the Cultural Affairs, 
School of E-learning as an example and we integrate the 
above factors such as the Duncan MacRae policy indicators, 
the Donald Kirkpatrick pattern, and the ROI index to seek an 
appropriate evaluation mode of governmental E-learning sites. 
At the same time, we also try to analyze the cost structure and 
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the profitability to which we can refer in our budget 
distribution, our cost control and the following added- value 
application. Our findings may help in the check and 
amendment of the annual plan. They can also serve as 
references for concerned evaluation of other kinds of 
E-learning websites. 

II. LITERATURE REVIEWS 

E-learning is a term that means something different to 
almost everyone who uses it. Some use it to refer to 
pack-aged content pieces and others to technical 
infrastructures. Some think only of asynchronous self-study 
while others think E-learning can encompass synchronous 
learning and collaboration. Almost all agree that E-learning is 
of strategic importance. 

E-learning is typically defined as learning using a 
computer and the Internet. Synonyms and/or related terms 
include learning that is: online, virtual, web-based, 
technology-assisted, distributed, open and distributed learning, 
and tele-learning. Related terms include distributed learning, 
computer-assisted learning, distance education, blended 
learning, computer-managed learning, learning management 
systems, and others. 

Therefore, what is the exactly meaning for E-learning? 
NCSA (the National Center for Supercomputing Applications) 
E-learning group provided a general definition: "E-learning is 
the acquisition and use of knowledge distributed and 
facilitated primarily by electronic means. This form of 
learning currently depends on networks and computers but 
will likely evolve into systems consisting of a variety of 
channels (e.g., wireless, satellite), and technologies (e.g., 
cellular phones, PDA's) as they are developed and adopted. 
E-learning can take the form of courses as well as modules 
and smaller learning objects. E-learning may incorporate 
synchronous or asynchronous access and may be distributed 
geographically with varied limits of time. "[4]. 

Rosenberg says, "E-learning refers to the use of internet 
technologies to deliver a broad array of solutions that enhance 
knowledge and performance. It is based on three fundamental 
criteria: 

• E-learning is networked, which makes it capable of 
instant updating, storage / retrieval, distribution and sharing of 
instruction or information. 

• It is delivered to the end user via a computer using a 
standard internet technology. 

• It focuses on the broadest view of learning-learning 
solutions that go beyond the traditional paradigms of 
training." [3]. 

In Taiwan, due to the "E-learning Park" of "the National 
Science and Technology Program for E-learning" promoted 
by the National Science Council, Executive Yuan. Those 
programs have been using "E-learning" as the program formal 
definition. Therefore, this research used the same definition. 

A. The Development of E-learning 

E-learning present four advantages: low cost of teaching 
and training, elevation of learners' interest, association of 
knowledge learning and new know-how, and more interactive 
channels and more flexibility. [2] In view of them, in 
developed countries, establishments such as governments, 



schools, and enterprises all allocate funds and resources to 
encourage research in learning and educational training. In 
Europe, the Commission of the European Communities 
announced the guideline of E-learning policy "The E-learning 
Action Plan-Designing tomorrow's education" in 2001. [5] In 
2002, The Secretary of Commerce in the United States 
addressed "2020 Visions-Transforming Education and 
Training through Advanced Technologies" report. The report 
was written by experts and scholars from Microsoft, HP, 
Harvard University, U.C. Berkley, Carnegie Mellon 
University, etc [6], Within this framework, our Executive 
Yuan passed the National Science and Technology Program 
for E-learning and decided to allocate four billion NTD to 
promote this project in the five following years. In the 
"Challenge 2008: National Development Plan" conceived by 
the Council for Economic Planning and Development, this 
National Science and Technology Program for E-learning is in 
the first part of the " E-life initiative" which is in its turn a 
part of the " E-Taiwan Program". 

Against that great conception, it was not rare that efforts 
made by industries, governments and academic 
establishments ended in failure. One of such failures was the 
Fathom. Com, a commercial teaching program in which 
Columbia University invested 30 million US dollars. The 
collaboration with world-famous establishments like the 
University of Michigan, the University of Chicago, the 
Cambridge University Press, the London School of 
Economics and Political Science and the British Museum in 
curricula and fund was not able to save Fathom from the 
elimination system of the market. In 2003, Columbia 
University decided to stop the investment program after four 
years of running [7]. (The Fathom. Com is still running and 
exploring new patterns.) Due to a large number of 
unsuccessful cases and cost augmentation, many people have 
begun the reflection on the effectiveness evaluation. 
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Fig. 1 Frame of policy indicators of MacRae 

B. The Effectiveness Evaluation of Public Policies 

The object of the present research is governmental 
E-learning website. Thus, before discussing the problem of 
the concerned effectiveness evaluation, we should outline a 
systematical frame suitable to the effectiveness of government 
policies. The "Duncan MacRae policy indicators" (Fig. 1) 
may be a credible frame. According to the definition of 
Duncan MacRae, policy indicators are public statistic values, 
which can be adopted as tools of public policy evaluation. 
Policy indicators include three end values, namely net 
economic benefit, subjective well-being and equity. What can 
be calculated in money is in general the net economic benefit. 
Its most typical example is the cost-to-benefit ratio of a given 
policy. The subjective well-being consists in the people's (or 
some concerned people's) satisfaction or pleasure with a 
given policy. In other words, the economic well-being lies 
with the market value while the subjective well-being is a 
feeling. The third end value is the distributional equity, which 
is not the total integration of social well-being but the 
distribution of social well-being [1], 
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Fig. 3 Spectrum of evaluation perspective (Source: William Horton, Evaluating E-learning, p7) 



C. Donald Kirkpatrick's Four-level Evaluation Pattern 

Hence on the subject of the policy, we can evaluate the 
effectiveness of governmental E-learning websites with the 
above frame of policy indicators, which include net economic 
benefit, subjective well-being, and equity. The economic 
four-level pattern proposed by Donald Kirkpatrick in 1959 is 
used to evaluate the concrete teaching and training 
effectiveness in more details. The four levels are as follows: 

(1) What is the reaction of the learner; (2) Has the learner 
get the knowledge or technique he or she wants to learn; (3) 
Can the learner apply what he or she has learned to his work; 
(4) Does the training achievement exert a positive influence 
upon the organization. 

The first level, the reaction evaluation, is a regular method 
most organizations adopt to know about the learner's feeling 
for the courses such as whether he or she likes them and 
whether they are practical. At the end of the training, the 
training organization will distribute questionnaires for an 
inquiry. The second level consists in seeing whether the 
learner has got what he or she wants to learn and whether he 
or she has attained the level he or she wants to attain. The best 
method is to test and do some practical operations. The third 
level is the performance. The ultimate goal of the training is 
the improvement of the learner's ability to apply what has 
learned in practical workplace. The evaluation of this level 
requires collaboration and observation of directors and 
colleagues. The last level is the balance evaluation. 



Economically speaking, the last profit should be larger than 
the initial cost. Thus, it is indispensable to check whether the 
output value and the profit of the organization are increased 
[3]. In the present research, we will try the above mode as the 
measuring tool for the governmental E-learning websites. 
However, the governmental training is divided into two parts 
according to its program's nature and aim: public 
functionaries and ordinary citizens. In this respect, we should 
allow for the different training aims and items of each group. 

Another point of view we propose through this research is 
that, while the main participants of the E-learning 
organization include designers of the teaching materials, 
teachers, managers and policy-makers, we should, on the 
basis of Kirkpatrick pattern, make different evaluations 
according to their different roles and foci of attention. 
E-learning developers (instructional designers, simulation and 
virtual environment engineers, software engineers) [6] may 
face to problems like media design and technological 
application. One of the characteristics of the E-learning is that 
the teaching materials are implicated by the integration of 
components like writings, graphics, cartoons, films, and 
sounds. The same materials, by grace of different software, 
can be supported by Internet, CD-title, mobile phone, PDA, 
etc. The production of different components requires different 
engineers, different technologies, different equipments, 
different costs, and different time. The learner's media 
effectiveness and response to each component are all in the 
focus of attention of the designers and developers. As to gifted 
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lecturers & experts and teachers & tutors, their attention is 
focused on effective course designs and learners' achievement. 
Managers and administrators are concerned with the 
program's achievement. CEO or policy makers insist on the 
whole training achievement and profitability. Hence we think 
it necessary to amend the Kirkpatrick pattern, as shown in Fig. 
2. 

D. Horton's Spectrum of Evaluation Perspectives 

According to different levels and aims, an evaluation can 
be carried out in different perspectives ranging from micro 
views to macro views. A spectrum of these views was 
elaborated by Horton [8], as shown in Fig. 3: 

E. ROI 

Enterprises' motivation is to reduce their cost by 
promoting E-learning. But more and more enterprises 
consider it as an investment plan and require teaching and 
training services to quantify E-learning in order to prove its 
achievement and effectiveness. Of course, there are 
E-learning effectiveness indicators like cost efficiency, 
learners' satisfaction and learning resources [9]. But because 
many unsuccessful cases are due to inappropriate financial 
planification, more and more directors of training services 
regard cost as the key element of E-learning achievement [11]. 
The cost-to-benefit ratio and the return of investment ratio are 
two financial tools that enterprises usually assume to measure 
the cost-to-benefit ratio [12]. They are also economic 
indicators proposed by MacRae to evaluate public policies 
and tools chosen by Kirkpatrick to quantify teaching or 
training achievements. In this research, we seek a good 
method to calculate the cost-to-benefit ratio and the return on 
investment ratio. 

The cost-to-benefit ratio is obtained by quantifying the 
total benefit and cost implicated in teaching and training. 
Then the ratio divides quantified benefit with quantified cost. 
The ROI is obtained by subtracting the cost from the total 
benefit. Then the ROL divides the obtained net benefit with 
the cost multiplied by the number 100[9]. They are formulated 
as follows: 

CBR = program benefits/program costs 

ROI (%) = Net program benefits/program costs X 100 

F. Cost Structure of E-learning Websites 

If we want to calculate CBR and ROI, we should above all 
know the full costing implicated in E-learning. And if we 
want to calculate the full cost, we would better make an 
analysis founded on the plan's lifespan [13]. The lifespan of 
an E-learning website can be distributed into five phases, 
namely capital investment, teaching material elaboration, 
advertisement, promotion, teaching research, and 
administrative service. Capital investment mainly implicates 
hard and software infrastructure, construction cost, and staff 
cost. Hardware includes server, networked storage appliance 
and load balancer. Software includes operation system, server 
system, media server system, data system, LMS, courseware 
tool license and frequency band rent. Teaching material 
elaboration needs hardwares like photographic appliance, 
non-linear montage apparatus, and image picking apparatus, 
scanner, microphone system and monitor while the relative 
softwares are also required for montage, multimedia 
production and cartoon fabrication. And the production staff 
cost should also be taken into account. 



The research is not done for free either. Hourly work, 
communication, teaching materials, place, and staff 
(administrators and assistants) all absorb a lot of cost. 
Promotion cost is mainly generated by advertisements, 
publicity prints, publicity activities, etc. The administration 
cost is generated by water and electricity consumption, office 
maintenance and office equipment, etc. But soft-hardware 
infrastructure is a part of capital investment and server's 
frequency band can be shared by other programs. So what are 
taken into the teaching and training cost calculation are only 
items like teaching material production, promotion, teaching 
and service 

G. Benefit Analysis of E-learning Websites 

Quantification of teaching and training benefit is always 
one of the challenges that human resources experts and 
scholars should face. Horton distributes the benefit into three 
classes: hard benefits, soft Benefits, and fuzzy benefits. Hard 
benefits are principally things like visible expenditure 
reduction, production increase, time decrease or quality 
melioration, which are easy to be quantified in money and 
immediately related to teaching and training. In general, soft 
benefits can be quantified only in indirect ways and this class 
covers performance, innovation, new conception, work 
environment amelioration, and employees' proficiency. Fuzzy 
benefits are intangible factors difficult to quantify such as 
satisfaction, cohesion, directorship's charisma, staff's service 
attitude and quality [8]. 

III. INTRODUCTION TO CASE 

CASE was opened in March 2002. Its objectives are as 
follows: (1) Establishment of software and hardware 
(websites, teaching materials) for E-learning; (2) Elaboration 
of digital cultural teaching materials; (3) Supply and transfer 
of cultural knowledge; (4) Increase in cultural population; (5) 
Formation of cultural bodies and virtual communities. The 
following tasks need to be accomplished. 
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Fig. 4 Platform/System of case 

A. Construction of a Teaching Platform/System 

This platform is a Learning Management System (LMS). 
It has the following functions: (1) Teaching management 
including contents of courses, teaching data, collection of 
homework, experience sharing, result management and 
announcement; (2) Course management including contents of 
courses, management of teaching materials, announcement, 
questionnaire and management of teachers and students; (3) 
Functions concerning teaching staff like "the same teacher" 
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function, "new course" function and "new comer" function, 
which is an assistance to E-leaming activities and can 
improve teaching and learning efficiency, as shown in Figure 
4. 

B. Programming of Digital Teaching Contents 

The "Basic and Advanced Courses on Community 
Culture" of the first and the second sessions attracted 110 000 
visitors or so, among whom over 2,600 were enrolled. This 
marked the first step forward in the course of digital learning 
promotion. Sustainable development in other cultural areas is 
also expected. And the objective, so to speak, enhancing the 
general cultural and artistic quality of the nation, will be at 
last attained. 

The courses on "Cultural Creative Industry" of the third 
session were very popular. The promotion of cultural creative 
industries is an important part of "Challenge 2008-Council for 
Economic Planning and Development Plan" of the Executive 
Yuan. Its goal is to propose an integrated system which may 
cater to different cultural or artistic industries in fields like 
staff training, research & development, information 
integration, financial aid, space supply, industry and 
university cooperation interface, marketing & promotion and 
tax exemption. Through E-learning, talents of the whole 
nation in creative industries can gather themselves together 
and push those cu local and central authorities. The courses 
attracted 165,000 visitors and the number of the enrollees 
amounted to over 2,600. 

The fourth session covered such units as "What Is the 
Culture of Taiwan", "Features of the Culture of Taiwan", 
"Taiwan's Repository of Cultural Heritage" "Beauty of 
Taiwan's Traditional Arts", "Ecologic Beauty of Taiwan," and 
"Community Culture in Taiwan". Through the courses, the 
students know better the culture of Taiwan and its future 
orientation. The knowledge about our cultural heritage has 
also aroused their thought. The success of this session proved 
that of the others. 600,000 visitors browsed our site with over 
20,000 enrolled. 




Fig. 5 Webpage of the CASE 

C. A/V Multimedia and Electronic Journal 

This allows CASE to become a teaching website with 
three dimensional functions. In addition to video streaming 
(VOD) courses, there are CASE TV units. Every week, 
campus headlines, key points of the courses and local news 
are produced and published, as shown in Figure 5. Numerous 
photos of the community are also taken and sent to the 
students. Those are part of the efforts exerted to make CASE 



not only a learning website but also a lieu of communication 
and exchange within a community. 

D. Blended Teaching 

CASE has its special teaching mode, which is blended 
teaching or in other words blended learning. That means 
online teaching and actual face-to-face activities like opening 
ceremony and completion ceremony and community meetings 
are blended. Face-to-face teaching (community craft for 
example) and online communication (bbs, instant message, 
voting, etc.) are also blended together. The flexible teaching 
mode permits a multiple way of displaying and sharing 
knowledge, information, and experience. Both explicit and 
tacit knowledge can be efficaciously managed by teachers and 
optimally absorbed by students. 

CASE has organized four sessions of courses. Its blended 
teaching, on the one hand, resorts to the network and on the 
other hand is dependent on actual study, actual lectures, actual 
studios, and actual visits. The actual and the virtual are well 
blended: students can get a better feeling for the context. The 
mixture of culture and technology makes art and culture more 
contagious. 

E. Criterion for Course Completion 

All the students of CASE cannot get their certificates of 
completion. Here are the criteria for completion: first, for each 
subject, no less than eighteen visits must be affected; second, 
for each subject no less than two comments or two answers 
should be published on BBS; third, homework must be done 
and turned in once a subject at least; fourth, questionnaires 
must be turned in; and last, report on learning experience with 
CASE must be turned in. 

IV. BENEFIT EVALUATION OF CULTURAL AFFAIRS SCHOOL OF 
E-LEARNING 

In the present research, we try to avail ourself of 
MacRae's policy indicators, Kirkpatrick's four-level 
evaluation pattern, and the ROI tool to analyze the 
effectiveness of Cultural Affairs, School of E-learning. The 
following figure shows us an integrated seven-item evaluation 
frame, which may serve as a reference to the future E-learning 
website appraisal. 

The analysis is based on the balance report of Cultural 
Affairs, School of E-learning, on the satisfaction inquiry and 
on the log statistics. For the first session, the number of the 
enrollees was 1,419 and for the second, it was 1,225. The 
students' sex, age, residence and education level are 
demonstrated as follows in the Table 1. We have also referred 
to Taiwan's internet use record in the year 2002 kept by the 
national portal net www.yam.com, in order to make a user 
distribution comparison. 

A. Equity Criterion Reached by Cultural Affairs, School of 
E-learning 

Students of Cultural Affairs, School of E-learning come 
from 21 cities and counties, offshore isles included, and there 
are also oversea students. The students are also of different 
education levels and ages. These data indicate that the 
Cultural Affairs, School of E-learning has reached the equity 
criterion, one of MacRae's policy indicators. It is a criterion 
difficult for traditional teaching and training organizations to 
attain. And three points deserve more concern: there are more 
female students than male students; the students are mostly 
aging from 31 to 40 years, but students aging 41-50 years also 
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Fig. 6 Integrated benefit evaluation frame for governmental e-learning website 



occupy a large proportion, which means that the E-learning 
population and the internet using population are the same 
group; students who have at least a master's degree occupy a 
high proportion, which is a good signal of the country's 
innovational potential. 

B. Satisfaction with the Cultural Affairs, School of E-learning 

The subjective well-being can be measured on the basis of 
the students' satisfaction probe. The satisfaction probe is made 
in the form of questionnaires distributed to the enrollees. The 
questionnaires should contain items on: curricula, platform 
operation, educational administration, and tutorship, website's 
typography, E-journal and general satisfaction extent. The 
items are to be detailed as follows : (1) curricula which 
includes teachers, course design, course content, 
supplementary material, and related general satisfaction extent; 
(2) platform operation which includes visual design, 
functional mechanism, operation facility, and related general 
satisfaction extent; (3) educational administration and 
tutorship which includes online teachers, web chief, service 
for clients (phone, email, etc.), and related general satisfaction 
extend ; (4) website typography which includes visual design, 
content, activities, interaction, and related general satisfaction 
extent ; (5) e-journal which includes typography, content 
compilation, and related general satisfaction extent ; (6) final 
general extent of satisfaction with CASE. The result is as 
follows(Table 2) 

The above figure shows that most students are satisfied 
with CASE and that the general satisfaction degree is over 
90/%. We can deduce that people can accept the pattern of 
E-learning. The probe is not reproachless because we have no 



means to get acquainted with the advice of those who did not 
receive the questionnaires. Hence in the future we should also 
take into account the proportion of tested students and that of 
the non-tested. Another phenomenon that attracts our attention 
is that the general satisfaction degree is higher than that of any 
individual one, which shows that one of the advantages of the 
school of E-learning lies in the synergy of its components. 

C. The Quantification Indicators of CASE 

The net economic benefit of E-learning is usually 
appraised on the ground of the output, the ROI, and 
employees' individual performance. However, culture and art 
dominate CASE'S teaching and training. Learning 
effectiveness in the cultural and artistic area is difficult to 
quantify because that: first, students' post learning 
performance is not the indicator of the learning effectiveness 
in this field; second, in general, the curricula concerns 
thinking and conception construction, which is not suitable for 
appraisal through a simple examination system; third, CASE 
is a governmental but not commercial organization and most 
of the students are also from non-commercial organizations, 
so the real-valued benefit is hard to measure. On the basis of 
Kirkpatrick's pattern and Horton's spectrum, we try to 
propose the following evaluation items and quantification 
indicators (as shown in Table 3): 

The correlation of different columns in Figure 3 is just one 
of the hypotheses we put forward at the end of the present 
research and it needs to be justified to be the basis of the 
coming research. The correlation is not absolute but relative. 
For website designers, the emphasis is laid upon the 
typography or the visual aspect of the page and attention is 
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paid to visitors' sensation or page's playfulness while the ease 
of use of the system platform counts mostly to programmers. 
However, that does not mean that between designers of 
different domains there is no interaction or exchange. Simon 
Lu, on the ground of Technology Acceptance Model, found 
that a learner's cognition and attitude reflects real time and 
frequency of his platform use. On the subject learning 
effectiveness, Kirkpatrick's evaluation pattern resorts mainly 
to test, examination, or practical operation. Yet since 
E-learning is in principle a system assumed to promote 
education to the whole society, it is unrealistic to implement 



an online impartial examination system. Today the 
constructivism of interactive discussion dominates the 
E-learning pedagogy and accordingly we adopt the number of 
times of discussions as our evaluation indicator. At last, after a 
comprehensive calculation and balance of the items of the 
above figures, we can get the last quantification. What remain 
to be done in the future research are the establishment of their 
mutual relationship and the determination of their relative 
response value. 

On the ground of the Table 3, we try to build the 
quantification data of CASE as follows in Table 4: 



TABLE 1 STUDENTS STATISTICS, CULTURAL AFFAIRS, SCHOOL OF E-LEARNING 





1st session 


2sd session 


Internet use of 
2002(source from 
www.yam.com) 


Remark 


Valid data 


1419students 


1225students 






Sex proportion 










M 


42% 


39% 


49.6% 




F 


58% 


61% 


50.4% 




Age 










Below 25 years 


12% 


19% 


40.4% 


In 2002 internet user age is below 24(included) 


Between 26-30 years 


19% 


23% 


26.2% 


In 2002 internet user age is 25-29 


Between 31-40years 


27% 


31% 


25.6% 


In 2002 internet user age is 30-39 


Between 41-50 years 


19% 


17% 


6.2% 


In 2002 internet user age is 40-49 


Over51 years 


5% 


7% 


1.6% 


2002 internet user age is over 50(included) 


Nil 


19% 


3% 






Region 










Ilan-Keelung 


3% 


2.77% 


2.95% 


Ilan and Keelung included 


Taipei City -Taipei County 


40% 


44.79% 


40.9% 


Taipei City and Taipei County included 


Taoyuan-Hsinchu-Miaoli 


12% 


7.97% 


14.1% 


Taoyuan City and County included, Hsinchu City and County 
included, Miaoli County included 


Taichung-Changhua 


15% 


16.28% 


15.4% 


Including Taichung City and County, Changhua County and 
Nantou County 


Yunlin-Chiayi-Tainan 


11% 


10.10% 


10.2% 


Including Tainan City and CountyChiayi County and Yunlin 
County 


Pingtung-Kaohsiung 


13% 


9.61% 


13.0% 


Kaohsiung City and County included, Pingtung County 
included 


Hualien-Taitung 


3% 


3.09% 


1.9% 


Hualien County and Taitung County included 


Offshore isles 


1% 


1.22% 


0.4% 


Including Chinmen County, Lienchiang County and Penghu 
County 


Other regions 




0.81% 




Other countries 


Nil 




3.34% 






Education 










Below senior high school and 
vocational school (included) 


7% 


8% 


21% 




Junior college and university 


76% 


70% 


69.3% 


Junior college included 


Postgraduate school 


16% 


19% 


9.7% 




Nil 




3% 







TABLE 2 STUDENTS' SATISFACTION WITH CASE 





Very satisfied 


Satisfied 


Indifferent 


Unsatisfied 


Very unsatisfied 


With CASE 


24 


67 


9 








With educational administration and 
tutorship 


23 


64 


12 


1 





With website design 


17 


63 


18 





1 


With curricula 


16 


73 


9 





1 


With E-journal 


16 


60 


24 








With platform operation 


11 


61 


24 


5 






Valid questionnaires: 89 copies 
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TABLE 3 GOVERNMENTAL E-LEARNING WEBSITE EVALUATION INDICATORS 



Kirkpatrick's four-level 
pattern 


Participants 


Horton's spectrum 


Objective indicators 


Subjective indicators 


Reaction 


Designers 


Curriculum 
components 


Online time of the components 


Satisfaction with the website design 


Platform 
operation 


Frequency of (the teaching material) utilization 


Satisfaction with the platform 


Learning 


Teachers 


Course content 


Number of entries in the course 


Satisfaction with the curricula 


Teaching and 

tutorshop 


Number of academic discussions 


Satisfaction with teachers 


Performance 


Managers 


Specific plan 


Enrollee population(person-time) 


General satisfaction 


Specific 
organization 


Number of completers (person-time) 


Students' performance result 


Results 


Policy makers 


Specific domain 


ROI 


Merits and favorable mentions 


Social status 


Volume of flow of the website 


Fame 



TABLE 4 EVALUATION INDICATORS OF CASE 



Sessions 


1st 


2nd 


Remark 


Items 


Number of 

times 


Average A 
Round 


Average B 
Round 


Number of 

times 


Average 

A 
Round 


Average 

B 
Round 


Average A is number of times/number of 

disciplines 
Average B is number of times/Completers 

4 disciplines for 1st session - 
8 disciplines for 2 nd session 


Number of times of 

teaching material 

download 


13,889 


3472 


53 


11,643 


1455 


29 


What we call download here concerns only 
supplementary teaching materials. 


Number of entries in 
the courses 


Nil 


Nil 


Nil 


41,018 


5127 


103 


Each login as an entry in course 


Number of 
discussions 


4,173 


1043 


16 


7,635 


955 


19 


Discussion means messages left on the BBS. 


Number of students 


1,419 


355 




1,228 


154 




Only enrollment and entry in course can 
justify status of student 


Number of 
enrollments 
(person-time) 


5,676 


1,419 




7,535 


942 




One student can be enrolled for several 
disciplines at one time. 


Number of 

completers 

(person-time) 


262 


66 




405 


51 




The calculating unit is person-time. 


Completion 
rate(person-time) 


0.05 


0.05 




0.05 


0.05 




Students should who comply with the 
criterion for completion. 


Volume of flow of 

the website 

(Number of visits 

(person-time)) 


60,000+ 






110,000 






The visit of the homepage is counted as a 
person-time. 



The Table 4 tells us that a standard learning management 
system supplies us with a standard daily record and statistics, 
which will be very helpful to the evaluating work. But the 
data above mentioned are not complete yet. Between the 
population of enrollments and the population of completers 
there are differences originated in the course of management 
strategy. Altogether, it is difficult to make a reasonable 
comparison at present. However, the publicity effect is not 
bad. For each session, there were always more than a 
thousand applications. And the balance of CASE indicates 
that since its opening (between the period 15 march 2002 and 
16 march 2003), its website volume of flow broke through 
190,000 person-times. 235 of the students of the first session 
renewed their application and continued their learning. Two 



interesting phenomena interest us. The percentage of the 
population who completed their learning was relatively low: 
only a percent. That was evidently related to the nature of 
CASE as a social education organization, which is usually not 
exigent with its students. But the increase of participation and 
completion rate requires innovative teaching strategy and 
course design too. In the second session the volume of flow 
was elevated, which shows that the accumulative 
effectiveness of E-learning is greater than that of traditional 
teaching. 

D. Positive Reaction of CASE 's Students 

During the sessions, an association of fellow students 
from seven regions was launched. With the initiation of the 
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TABLE 5 ROI OF CASE 












Total budget 
(NT) 


Number of 
enrollees 
(person) 


Population 

Of 

completers 

(person) 


Per capita 

teaching cost 

(A) 


Per capita 

teaching cost 

(B) 


ROI 

(A) 


ROI 
(B) 


1st session of E-learning 


1,996,000 


1,419 


262 


1,407 


7,618 






2 nd session 


1,858,800 


1,228 


405 


1,514 


4,590 






Average of E-learning School 








1,461 


6,104 


535% 


128% 


Traditional teaching and 
training (A) 


2,016,300 


200 


200 


10,082 


10,082 






Traditional teaching and 
training (B) 


720,000 


130 


130 


5,538 


5,538 






Average of traditional training 








7,810 


7,810 







* Traditional teaching and training (A) To take the Budget for "WebPages Elaboration Class for Cultural Functionaries" of Cultural Affairs Council of 2000 for 
reference Traditional teaching and training (B) To take the Budget for "Class for Cultural and Artistic Administrators" of Cultural Affairs Council of 2003 for 

reference 
* Per capita teaching cost (A)= Total budget /Number of students to admit 

* Per capita teaching cost (B)= Total budget / population of completers (person-time) 

* ROI Cost (A)=Average of E-learning School (A)/Average of traditional training (A) 

* ROI Cost (B)= Average of E-learning School (B)/Average of traditional training (B) 



enterprise strategy studio, seminars on community building 
were organized and related knowledge obtained from CASE 
resulted in five practical programs for community building. 
Another example was the documentary about Ouches. The 
idea came from the BBS of CASE. The problem of offshore 
isles was once the focus of discussion in CASE'S online 
forum. Ouchiu was a very typical example of this kind of 
problem. Then the local people and CASE'S community 
collected films and files on the isle. An exhibition was also 
organized for this. All those are things that cannot be 
quantified and appraised in money. 

E. ROI of CASE 

In the end, we treat the ROI evaluation. Before the 
opening of E-learning schools, teaching and training was a 
relatively concrete activity that requires students of different 
cities or counties to come to a given place to study. To 
students, travel and the lodgment are extremely expensive and 
cost of the training organization is also very high. But to 
Rosenberg, what E-learning spares students is not simply the 
travel or the lodgment expenses but the opportunity cost [3]. 
For traditional teaching as well as for E-learning, cost 
calculation, including the opportunity cost calculation makes 
a difference only in some individual cases. And in 
non-commercial public services, the comparison is even more 
difficult to draw [14]. Consequently, in the present research, 
as the basic principle of ROI and cost-to-benefit evaluation, 
we prefer the analysis of the per- capita learning cost to the 
traditional theory, namely, that regards cost as a benefit 
indicator, as shown in Table 5. 

In the budgets of the first and second sessions of CASE, 
no equipment expense was involved and all the costs were 
related to staff, service, and administration. In this respect, a 
comparison can be drawn between it and traditional training. 
From Figure 7, we can deduce the difficulty in proving that 
the ROI of E-learning is evidently larger than that of 
traditional training. Because though the difference in 
education is large and training cost, actual or virtual, is 
accordingly determined by nature of curricula, teaching 



objective and hour. So Rosenberg was right in saying that the 
decisive motivation of E-learning consists in sparing time and 
not in sparing money. The teaching materials of E-learning 
can be repeatedly used and the transmission time and cycle 
may be shortened. That is also what traditional training can in 
no case realize. 




Fig. 7 curriculum schedule of case 
V. CONCLUSION 

In the present research, we have taken CASE to treat upon 
the problem of governmental E-learning website effectiveness 
evaluation and we find that E-learning can help improve the 
equity related to resource distribution and reduce disparity 
between towns and country. People are also satisfied with 
E-learning experience. Economically, E-learning does not cost 
more than traditional training and if we take into account 
elements like time cost, opportunity cost, teaching material 
recycle and policy publicity, E-learning takes on more 
advantages. But on the subject of planning, we have neither 
common standard nor standard statistics on system platform, 
courseware design, teaching material presentation, and 
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teaching appraisal, which makes it arduous to confront 
different plans, different course wares, and different 
components. Even subjective evaluation indices have much to 
be desired. The SCORM (Sharable Content Object Reference 
Model) promoted by government is good for both courseware 
exchange and regulation on evaluation indices. The 
policy-making support system may be further established. 

In a school of E-learning that fails actual control that 
characterizes traditional training to raise completion rate, 
some work needs to be done on didactics, pedagogic strategy, 
teaching material design and evaluation mode. We have also 
found that in E-learning there are some zealots. The epithet is 
not pejorative and the phenomena should be a part of our 
future research program. CASE'S teaching and training cover 
a considerable variety of activities: media flow, dynamic 
briefs, supplementary teaching materials, interactive 
discussions, instant messenger (icq), E-journal, campus 
headline, and CASE TV. However, statistical insufficiency 
impedes an exact evaluation of its benefit. Students' 
satisfaction extent probe signifies synergetic effectiveness of 
different components, which is yet to be studied. 

To conclude, we affirm CASE'S positive effect upon the 
promotion of cultural policies and the building of internet 
culture, because most of CASE'S students of the first and 
second sessions were adults from society. Their zeal was 
corroborated by the number of enrollees, the volume of flow, 
the number of discussions, etc. For the present phase, the 
governmental E-learning website is a subprogram of the 
National Science and Technology Program for E-learning. In 
this respect, CASE contributes to technology diffusion. 
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